Depth map generation device capable of correcting occlusion

ABSTRACT

A depth map generation device capable of correcting occlusion includes at least two image capture pairs and a depth map generator. The at least two image capture pairs is used for capturing a plurality of images. The depth map generator is coupled to the two image capture pairs for generating a first depth map and a second depth map according to the plurality of images, wherein when the first depth map includes a first occlusion region and a first non-occlusion region, the depth map generator corrects the first occlusion region according to the second depth map.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/479,331, filed on Mar. 31, 2017 and entitled “Camera with PanoramicImage and Depth Information and Depth Capturing Device and System,” andthe benefit of U.S. Provisional Application No. 62/549,967, filed onAug. 25, 2017 and entitled “System of depth occlusion correction andmethod thereof,” the contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a depth map generation device, andparticularly to a depth map generation device capable of correctingocclusion.

2. Description of the Prior Art

When an object exists between a stereo camera and another object, aregion of another object may be only shown on an image captured by aleft eye image capturer (a right eye image capturer) of the stereocamera. That is, the right eye image capturer (the left eye imagecapturer) of the stereo camera cannot sense the region because ofocclusion of the object. Therefore, when a depth map generator coupledto the stereo camera utilizes a triangulation algorithm provided by theprior art to generate a depth map corresponding to the another object,because the right eye image capturer (the left eye image capturer)cannot sense the region, the depth map will include an invalid areacorresponding to the region, wherein an issue corresponding to theinvalid area becomes worse with increase of a baseline between the lefteye image capturer and the right eye image capturer. Therefore, how todesign a depth map generation device capable of correcting occlusionwill become an important issue.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides a depth map generationdevice capable of correcting occlusion. The depth map generation deviceincludes at least two image capture pairs and a depth map generator. Theat least two image capture pairs are used for capturing a plurality ofimages. The depth map generator coupled to the at least two imagecapture pairs for generating a first depth map and a second depth mapaccording to the plurality of images, wherein when the first depth mapincludes a first occlusion region and a first non-occlusion region, thedepth map generator corrects the first occlusion region according to thesecond depth map.

An embodiment of the present invention provides a depth map generationdevice capable of correcting occlusion. The depth map generation deviceincludes an image capture module and a depth map generator. The imagecapture module includes at least two image capture pairs, and each imagecapture pair of the at least two image capture pairs is composed of twoimage capturers, or an image capturer and a light source. The depth mapgenerator is coupled to the image capture module. When a first depth mapof at least two depth maps generated by the depth map generatorcorresponding to the at least two image capture pairs includes anocclusion region and a non-occlusion region, the depth map generatorutilizes a second depth map of the at least two depth maps to correctthe occlusion region.

An embodiment of the present invention provides a depth map generationdevice capable of correcting occlusion. The depth map generation deviceincludes at least two image capture pairs and a depth map generator. Theat least two image capture pairs are used for capturing a plurality ofimages. The depth map generator is coupled to the at least two imagecapture pairs for generating a first depth map and a second depth mapaccording to the plurality of images, wherein the depth map generatorfurther executes at least one of correcting an occlusion region of thefirst depth map according to the second depth map and fusing the firstdepth map and the second depth map to generate a fused depth map.

An embodiment of the present invention provides a depth map generationdevice capable of correcting occlusion. The depth map generation deviceincludes two image capture pair groups and a depth map generator. Thetwo image capture pair groups are used for capturing a plurality offirst images and a plurality of second images, respectively. The depthmap generator is coupled to two image capture pair groups for generatinga plurality of first depth maps and a plurality of second depth mapsaccording to the plurality of first images and the plurality of secondimages respectively, and fusing the plurality of first depth maps andthe plurality of second depth maps to generate a first fused depth mapand a second fused depth map, respectively. When the first depth mapincludes a first occlusion region and a first non-occlusion region, thedepth map generator further corrects the first occlusion regionaccording to the second fused depth map.

The present invention provides a depth map generation device capable ofcorrecting occlusion. The depth map generation device utilizes acorresponding valid area of a second depth map of at least two depthmaps corresponding to at least two image capture pairs generated by thedepth map generator to correct (or substitute for) at least oneocclusion region included in a first depth map of the at least two depthmaps when the first depth map includes the at least one occlusionregion. Therefore, compared to the prior art, because the presentinvention utilizes the corresponding valid area to correct (orsubstitute for) the at least one occlusion region, the present inventioncan effectively solve a problem of a depth map including an occlusionregion.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a first embodiment of the presentinvention.

FIG. 2A is a diagram illustrating a region of a depth map correspondingto an object including an occlusion region.

FIG. 2B is a diagram illustrating a region of a depth map correspondingto an occlusion region of another depth map.

FIG. 3 is a diagram illustrating the first baseline and the secondbaseline not parallel to each other.

FIG. 4 is a diagram illustrating a depth map generation device capableof correcting occlusion according to another embodiment of the presentinvention.

FIG. 5 is a diagram illustrating the structured light being a codingpattern.

FIG. 6 is a diagram illustrating the depth map generator generating amirror depth map.

FIG. 7 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a second embodiment of the presentinvention.

FIG. 8 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a third embodiment of the presentinvention.

FIG. 9 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a fourth embodiment of the presentinvention.

FIG. 10 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a fifth embodiment of the presentinvention.

FIG. 11 is a diagram illustrating a depth map generation device capableof correcting occlusion according to a sixth embodiment of the presentinvention.

DETAILED DESCRIPTION

Please refer to FIG. 1. FIG. 1 is a diagram illustrating a depth mapgeneration device 100 capable of correcting occlusion according to afirst embodiment of the present invention, wherein the depth mapgeneration device 100 includes image capturers 102, 104, 106 and a depthmap generator 108, and the depth map generator 108 is coupled to theimage capturers 102, 104, 106. But, the present invention is not limitedto the depth map generation device 100 only including the imagecapturers 102, 104, 106, that is, the depth map generation device 100can include more than three image capturers. As shown in FIG. 1, a firstbaseline BL1 exists between the image capturer 102 and the imagecapturer 104, and a second baseline BL2 exists between the imagecapturer 104 and the image capturer 106, wherein the image capturers102, 104, 106 and the depth map generator 108 are installed on a printedcircuit board 112. But, for simplifying FIG. 1, only the image capturers102, 104, 106 are shown on the printed circuit board 112. In addition,in one embodiment of the present invention, a length of the firstbaseline BL1 is equal to a length of the second baseline BL2.

The image capturers 102, 104, 106 form two image capture pairs, whereinthe image capturers 102, 104 act as a first image capture pair, theimage capturers 106, 104 act as a second image capture pair, and boththe first image capture pair and the second image capture pair includethe image capturer 104. As shown in FIG. 1, the depth map generator 108is electrically connected to the image capturers 102, 104, 106 forgenerating a depth map corresponding to each image capture pair of thefirst image capture pair and the second image capture pair according toimages captured by the each image capture pair. As shown in FIG. 1, whena first object 101 exists between the depth map generation device 100and a second object 103, the image capturer 106 will not sense a region1032 of the second object 103 because a visual field of the imagecapturer 106 corresponding to the second object 103 is occluded by thefirst object 101, resulting in a right side of a region 1034 of a depthmap DP1 (corresponding to the second object 103) generated by the depthmap generator 108 according to images captured by the second imagecapture pair including the second object 103 including an occlusionregion ODP1 (as shown in FIG. 2A). As shown in FIG. 1, taking the imagecapturer 104 being installed on a left of the image capturer 106 and theimage capturer 104 being installed on a right of the image capturer 102as an example, a search direction of the depth map generator 108 withinimages captured by the image capturer 106 is toward left based on imagescaptured by the image capturer 104, wherein the occlusion region ODP1 ofthe depth map DP1 is an invalid area of the depth map DP1. In addition,the depth map generator 108 can also generate a depth map DP2corresponding to the first image capture pair according to imagescaptured by the first image capture pair including the second object103, wherein the depth map DP2 has a region 1036 corresponding to theobject 103, and the search direction of the depth map generator 108within images captured by the image capturer 102 is toward right. Asshown in FIG. 1, although the first object 101 exists between the depthmap generation device 100 and the second object 103, because visualfields of the image capturers 104, 102 corresponding to the secondobject 103 are not occluded by the first object 101, both the imagecapturers 104, 102 can sense the region 1032 of the second object 103.Therefore, based on the images captured by the image capturer 104, thesearch direction of the depth map generator 108 within the imagescaptured by the image capturer 102 is toward right to generate the depthmap DP2, so a region VDP (as shown in FIG. 2B) of the depth map DP2corresponding to the occlusion region ODP1 of the depth map DP1 is avalid area. Therefore, the depth map generator 108 can utilize depthinformation of the region VDP of the depth map DP2 corresponding to theocclusion region ODP1 of the depth map DP1 to correct (e.g. substitutefor) the occlusion region ODP1 of the depth map DP1, and then output acorrected depth map corresponding to the depth map DP1. But, in anotherembodiment of the present invention, the occlusion region ODP1 of thedepth map DP1 and a predetermined contiguous area within the depth mapDP1 adjacent to the occlusion region ODP1 are replaced with acorresponding valid area of the depth map DP2 to prevent the depth mapgenerator 108 from not effectively correcting the occlusion region ODP1of the depth map DP1 because an error of the occlusion region ODP1 ofthe depth map DP1, wherein a range of the predetermined contiguous areacan be adjusted by a user. In addition, because the length of the firstbaseline BL1 is equal to the length of the second baseline BL2, so thedepth map generator 108 can directly utilize the depth information ofthe region VDP of the depth map DP2 to substitute for the occlusionregion ODP1 of the depth map DP1. That is, when the depth map generator108 utilizes the depth information of the region VDP of the depth mapDP2 to substitute for the occlusion region ODP1 of the depth map DP1,any normalization is not executed on a disparity of the region VDP tomake the disparity of the region VDP match a disparity of the depth mapDP1.

However, in another embodiment of the present invention, if the lengthof the first baseline BL1 is not equal to the length of the secondbaseline BL2, when the depth map generator 108 utilizes the depthinformation of the region VDP of the depth map DP2 to substitute for theocclusion region ODP1 of the depth map DP1, normalization needs to beexecuted on the disparity of the region VDP, that is, the disparity ofthe region VDP needs to be multiplied by a normalization ratio to matchthe disparity of the depth map DP1, wherein the normalization ratio isdetermined by equation (1):

NRA=BL2/BL1  (1)

As shown in equation (1), NRA is the normalization ratio, BL1 is thelength of the first baseline, and BL2 is the length of the secondbaseline. Therefore, when the depth map generator 108 utilizes the depthinformation of the region VDP of the depth map DP2 to substitute for theocclusion region ODP1 of the depth map DP1, the disparity of the regionVDP is a normalized disparity. In addition, in another embodiment of thepresent invention, when the depth map DP1 and the depth map DP2 arerepresented by distance, because a unit of the distance utilized in thedepth map DP1 and the depth map DP2 is identical (e.g. meter), anynormalization conversion is not executed on the depth map DP2 to makethe depth map DP2 match the depth map DP1.

In addition, when the first baseline BL1 and the second baseline BL2 arenot parallel to each other (as shown in FIG. 3), the depth map generator108 further executes geometric calibration (e.g. rotation calibration)on at least one of the depth map DP1 and the depth map DP2. That is, thedepth map generator 108 can utilize a first rotation matrix to rotatethe depth map DP1, or utilize a second rotation matrix to rotate thedepth map DP2, or utilize the first rotation matrix and the secondrotation matrix to rotate the depth maps DP1, DP2 respectively, whereinthe first rotation matrix and the second rotation matrix are obtainedaccording to one of the first baseline BL1 and the second baseline BL2,or according to a reference line different from the first baseline BL1and the second baseline BL2, and the first rotation matrix and thesecond rotation matrix are used for making the first baseline BL1parallel to the second baseline BL2 after the depth map generator 108executes the geometric calibration on at least one of the depth map DP1and the depth map DP2.

In addition, in another embodiment of the present invention, the depthmap generation device 100 can include at least one light source (e.g.alight source 402 shown in FIG. 4), wherein the at least one lightsource is used for emitting at least one structured light. As shown inFIG. 4, the light source 402 is an infrared light source for emittingstructured light (or a random pattern), and the light source 402 is usedfor making better quality of the depth maps DP1, DP2 generated by thedepth map generation device 100, wherein as shown in FIG. 5, thestructured light is a coding pattern (corresponding to a randompattern). But, the present invention is not limited to light source 402being an infrared light source. That is, the light source 402 can beother type of light sources (e.g. the light source 402 can be a visiblelight source). Or, in another embodiment of the present invention, thedepth map generation device 100 can also include at least one infraredlaser light source. Taking the depth map generation device 100 as anexample, the light source 402 is turned on according to at least one ofluminance of an environment which the depth map generation device 100 islocated at, the quality of the depth map DP1 (or the depth map DP2, anda difference of the depth map DP1 (or the depth map DP2) correspondingto turning-on and turning-off of the light source 402.

When the light source 402 is turned on according to the luminance of theenvironment which the depth map generation device 100 is located at, acontroller (not shown in FIG. 4) can determine the luminance of theenvironment which the depth map generation device 100 is located ataccording to at least one of a shutter time, an exposure time, and anISO gain currently set by the image capturer 102 (or the image capturer104, or the image capturer 106). Taking the exposure time as an example,in one embodiment of the present invention, when the shutter time of theimage capturer 102 is fixed (or the image capturer 102 has no shutter),the controller can determine whether to turn on the light source 402according to a value GEX generated by equation (2):

GEX=gain*EXPT  (2)

As shown in equation (2), “gain” shown in equation (2) is the ISO gaincorresponding to the image capturer 102 and “EXPT” shown in equation (2)is the exposure time corresponding to the image capturer 102. When thevalue GEX is greater than a high threshold value, it means that theluminance of the environment which the depth map generation device 100is located at is too dark, so the controller turns on the light source402; and when the value GEX is less than a low threshold value, it meansthat the luminance of the environment which the depth map generationdevice 100 is located at is bright enough, so the controller turns offthe light source 402, wherein the high threshold value is greater thanthe low threshold value. In addition, when a maximum value of the valueGEX (corresponding to a maximum exposure time and a maximum gain of theimage capturer 102) cannot be always greater than the high thresholdvalue, the controller can turn on the light source 402 according tocurrent luminance of the environment which the depth map generationdevice 100 is located at.

When the light source 402 is turned on according to the quality of thedepth map DP1, the controller can determine the quality of the depth mapDP1 according to at least one of a number of pixels with invalid valueswithin the depth map DP1 and smoothness of the depth map DP1. Forexample, in one embodiment of the present invention, the controller candetermine whether to turn on the light source 402 according to a valueCOST generated by equation (3):

COST=a*mean(HPF(x))+b*invalid_cnt(x)  (3)

As shown in equation (3), “HPF(x)” shown in equation (3) corresponds toa response of a high pass filter (because the smoothness of the depthmap DP1 corresponds to high frequency areas of the depth map DP1),“mean(HPF(x))” shown in equation (3) corresponds to an average of theresponse of the high pass filter (but, in another embodiment of thepresent invention, “mean(HPF(x))” shown in equation (3) can be replacedwith a sum corresponding to the response of the high pass filter),“invalid_cnt(x)” shown in equation (3) represents the number of thepixels of with the invalid values, “x” shown in equation (3) representsthe depth map DP1, and “a, b” shown in equation (3) are coefficients.When the value COST is greater than a threshold value, it means that theluminance of the environment which the depth map generation device 100is located at is too dark or shot objects of the depth map DP1 have notexture, so the controller turns on the light source 402. In addition,after the light source 402 is turned on for a predetermined time, thecontroller can attempt to turn off the light source 402 and make theimage capturer 102 capture at least one image, and then the controllercalculates a cost value corresponding to the at least one imageaccording to equation (3). If the cost value corresponding to the atleast one image is still greater than the threshold value, thecontroller turns on the light source 402 again and executes the abovementioned operation again after the controller turns on the light source402 for the predetermined time; and if the cost value corresponding tothe at least one image is less than the threshold value, the controllerturns off the light source 402 until the cost value corresponding to theat least one image is greater than the threshold value again.

In addition, the controller can turn on and turn off the light source402, and determine the quality of the depth map DP1 according to thedifference of the depth map DP1 corresponding to turning-on andturning-off of the light source 402. If the difference of the depth mapDP1 corresponding to turning-on and turning-off of the light source 402is less than a reference value, it means that turning-on and turning-offof the light source 402 does not influence the quality of the depth mapDP1, so the controller can turn off the light source 402.

In addition, after the light source 402 is turned on, the controller canoptionally adjust intensity of the light source 402 according toluminance corresponding to a plurality of images captured by the imagecapturers 102, 104 and a target value, wherein the target value is setaccording to reflection coefficient of a human skin of the usercorresponding to the structured light emitted by the light source 402.For example, the controller can generate a luminance distribution mapcorresponding to the plurality of images according to the plurality ofimages, and optionally adjust the intensity of the light source 402according to a percentage of the depth map DP1 occupied by an areacorresponding to a maximum luminance value of at least one luminancevalue within the luminance distribution map greater than the targetvalue. In addition, in another embodiment of the present invention, thecontroller can generate average luminance corresponding to the pluralityof images according to the plurality of images, and optionally adjustthe intensity of the light source 402 according to the average luminanceand the target value. In addition, in another embodiment of the presentinvention, the controller can generate a luminance histogramcorresponding to a plurality of pixels of the plurality of imagesaccording to the plurality of images, and optionally adjust theintensity of the light source 402 according to a median of the luminancehistogram and the target value, or according to a quartile of theluminance histogram and the target value.

In addition, in another embodiment of the present invention, after thelight source 402 is turned on, the controller can optionally dynamicallyadjust the intensity of the light source 402 according to a distancebetween at least one predetermined object within the plurality of imagesand the image capturer 102 (or the image capturer 104, or the imagecapturer 106) and a first lookup table, wherein the first lookup tablestores relationships between a distance corresponding to an object andthe intensity of the light source 402. In addition, in anotherembodiment of the present invention, the controller can optionallydynamically adjust the intensity of the light source 402 according tothe distance between the at least one predetermined object within theplurality of images and the image capturer 102 (or the image capturer104, or the image capturer 106) and a first correlation formula.

In addition, in another embodiment of the present invention, thecontroller continuously detects the luminance of the environment whichthe depth map generation device 100 is located at under the light source402 being turned off. When the luminance of the environment is brighter,the controller increases the intensity of the light source 402 (when thelight source 402 is turned on) according to a second lookup table,wherein the second lookup table stores relationships between theintensity of the light source 402 (when the light source 402 is turnedon) and the luminance of the environment. In addition, in anotherembodiment of the present invention, when the luminance of theenvironment is brighter, the controller increases the intensity of thelight source 402 (when the light source 402 is turned on) according to asecond correlation formula.

In addition, in another embodiment of the present invention, after thecontroller first turns off the light source 402, the controller detectsthe luminance of the environment. Then, according to an automaticexposure (AE) algorithm well-known to one of ordinary skill in the art,the controller utilizes the exposure time (or at least one of theshutter time, the exposure time, and the ISO gain) of the image capturer102 (or the image capturer 104, or the image capturer 106) to make theluminance of the environment be reduced to not to interfere with theimage capturer 102 (or the image capturer 104, or the image capturer106), and fixes a current exposure time of the image capturer 102 (orthe image capturer 104, or the image capturer 106). Then, the controllerturns on the light source 402 and detects the intensity of the lightsource 402 until the intensity of the light source 402 is up to thetarget value.

In addition, in another embodiment of the present invention, a depth mapcan also be generated by the depth map generator 108 searching the samedirection. For the second image capture pair, based on the imagescaptured by the image capturer 104, the search direction of the depthmap generator 108 within the images captured by the image capturer 106is still toward left to generate the depth map DP1 (as shown in FIG.2A); for the first image capture pair, the depth map generator 108 firstmirrors the images captured by the image capturers 102, 104. Then, basedon mirror images corresponding to the image capturer 104, the searchdirection of the depth map generator 108 within mirror imagescorresponding to the image capturer 102 is toward left to generate amirror depth map MDP2 (as shown in FIG. 6), wherein the mirror depth mapMDP2 also include a region 1038 corresponding to the object 103, and themirror depth map MDP2 also include a mirror valid region MVDPcorresponding to the occlusion region ODP1 of the depth map DP1. Beforethe occlusion region ODP1 is corrected, for making coordinates of themirror depth map MDP2 match coordinates of the depth map DP1corresponding to the object 103, the depth map generator 108 first needsto mirror the mirror depth map MDP2 to generate the depth map DP2 (asshown in FIG. 2B), and then can utilize the depth information of theregion VDP of the depth map DP2 to substitute for the occlusion regionODP1 of the depth map DP1.

In addition, the depth map generator 108 can be a field programmablegate array (FPGA) with the above mentioned functions of the depth mapgenerator 108, or an application-specific integrated circuit (ASIC) withthe above mentioned functions of the depth map generator 108, or asoftware module with the above mentioned functions of the depth mapgenerator 108.

In addition, in another embodiment of the present invention, if thevisual field of the image capturer 102 of the first image capture pair(composed of the image capturers 102, 104) corresponding to an object isoccluded by a first object, depth information of a first depth mapgenerated by the first image capture pair corresponding to the objectwill include a first occlusion region and a first non-occlusion region,and depth information of the first occlusion region is an invalid areaof the first depth map; if the visual field of the image capturer 106 ofthe second image capture pair (composed of the image capturers 104, 106)corresponding to the object is occluded by a second object, depthinformation of the second depth map generated by the second imagecapture pair corresponding to the object will include a second occlusionregion and a second non-occlusion region, and depth information of thesecond occlusion region is an invalid area of the second depth map.Therefore, when the first non-occlusion region is a valid area of thefirst depth map corresponding to the second occlusion region and thesecond non-occlusion region is a valid area of the second depth mapcorresponding to the first occlusion region, the depth map generator 108can correct (e.g. substitute for) the second occlusion region and thefirst occlusion region according to the first non-occlusion region andthe second non-occlusion region, respectively.

In addition, please refer to FIG. 7. FIG. 7 is a diagram illustrating adepth map generation device 700 capable of correcting occlusionaccording to a second embodiment of the present invention, wherein thedepth map generation device 700 includes image capturers 102, 106, alight source 702, and a depth map generator 108. But, the presentinvention is not limited to the depth map generation device 700 onlyincluding the image capturers 102, 106. That is, the depth mapgeneration device 700 can also include more than three image capturers.In addition, the light source 702 and the image capturer 102 form afirst image capture pair, and the light source 702 and the imagecapturer 106 form a second image capture pair. As shown in FIG. 7, afirst baseline BL1 exists between the light source 702 and the imagecapturer 102, and a second baseline BL2 exists between the light source702 and the image capturer 106. The depth map generator 108 iselectrically connected to the image capturers 102, 106 for generating adepth map corresponding to each image capture pair of the first imagecapture pair and the second image capture pair according to an imageincluding the structured light captured by the each image capture pair.That is, the depth map generator 108 will generate two depth mapscorresponding to the first image capture pair and the second imagecapture pair. Therefore, when a first depth map corresponding to thefirst image capture pair includes an occlusion region and a second depthmap corresponding to the second image capture pair does not include anyocclusion region, or when an occlusion region included in the firstdepth map corresponding to the first image capture pair and an occlusionregion included in the second depth map corresponding to the secondimage capture pair correspond to different regions, the depth mapgenerator 108 of the depth map generation device 700 can utilize theabove-mentioned operational principles of the depth map generator 108 ofthe depth map generation device 100 to correct the occlusion regionincluded in the first depth map corresponding to the first image capturepair and the occlusion region included in the second depth mapcorresponding to the second image capture pair, so further descriptionthereof is omitted for simplicity. In addition, operational principlesof the light source 702 can also be referred to operational principlesof the light source 402, so further description thereof is also omittedfor simplicity.

In addition, please refer to FIG. 8. FIG. 8 is a diagram illustrating adepth map generation device 800 capable of correcting occlusionaccording to a third embodiment of the present invention, wherein thedepth map generation device 800 includes image capturers 102, 104, alight source 702, and a depth map generator 108. But, the presentinvention is not limited to the depth map generation device 800 onlyincluding the two image capturers 102, 104. That is, the depth mapgeneration device 800 can include more than three image capturers. Inaddition, the image capturers 102, 104 form a first image capture pairand the light source 702 and the image capturer 104 form a second imagecapture pair. As shown in FIG. 8, a first baseline BL1 exists betweenthe image capturers 102, 104, and a second baseline BL2 exists betweenthe light source 702 and the image capturer 104. As shown in FIG. 8, thedepth map generator 108 is electrically connected to the image capturers102, 104 for generating two depth maps corresponding to the first imagecapture pair and the second image capture pair according to imagescaptured by the first image capture pair and images including thestructured light captured by the second image capture pair. Therefore,the depth map generator 108 of the depth map generation device 800 canutilize the above-mentioned operational principles of the depth mapgenerator 108 of the depth map generation device 100 to correct a depthmap with an occlusion region, so further description thereof is omittedfor simplicity.

In addition, please refer to FIG. 9. FIG. 9 is a diagram illustrating adepth map generation device 900 capable of correcting occlusionaccording to a fourth embodiment of the present invention, wherein thedepth map generation device 900 includes image capturers 102, 104, 105,106, 107 and a depth map generator 108. But, the present invention isnot limited to the depth map generation device 900 only including theimage capturers 102, 104, 105, 106, 107. As shown in FIG. 9, the imagecapturers 104, 106 form a first image capture pair, the image capturers104, 107 form a second image capture pair, the image capturers 104, 102form a third image capture pair, and the image capturers 104, 105 form afourth image capture pair, and the fourth embodiment of the presentinvention is based on the first image capture pair. Therefore, if afirst depth map corresponding to the first image capture pair includesan occlusion region and a second depth map corresponding to the secondimage capture pair does not include any occlusion region, the depth mapgenerator 108 first needs to execute the geometric calibration on thesecond depth map based on the first image capture pair. That is, thesecond image capture pair will first be geometrically converted to acondition which can make the depth map generator 108 generate the seconddepth map. After the depth map generator 108 generates the second depthmap, the depth map generator 108 will rotate the second depth map (e.g.rotate the second depth map 90 degree) to make a baseline correspondingto the second image capture pair parallel to a baseline corresponding tothe first image capture pair. Similarly, if the first depth map includesthe occlusion region and a third depth map corresponding to the thirdimage capture pair does not include any occlusion region, the depth mapgenerator 108 first needs to execute the geometric calibration on thethird depth map based on the first image capture pair. That is, thethird image capture pair will first be geometrically converted to acondition which can make the depth map generator 108 generate the thirddepth map. After the depth map generator 108 generates the third depthmap, the depth map generator 108 will rotate the third depth map (e.g.rotate the third depth map 180 degree) to make a baseline correspondingto the third image capture pair parallel to a baseline corresponding tothe first image capture pair. But, in another embodiment of the presentinvention, the depth map generator 108 can also first mirror the thirddepth map to achieve an effect of the geometric calibration, whereinoperational principles of the depth map generator 108 mirroring thethird depth map can be referred to the embodiment shown in FIG. 6, sofurther description thereof is omitted for simplicity. In addition,operational principles of the fourth image capture pair can be referredto the above-mentioned operational principles of the second imagecapture pair, so further description thereof is omitted for simplicity.In addition, after the depth map generator 108 of the depth mapgeneration device 900 executes the geometric calibration, subsequentoperational principles of the depth map generator 108 of the depth mapgeneration device 900 can be referred to the operational principles ofthe depth map generator 108 of the depth map generation device 100, sofurther description thereof is omitted for simplicity.

In addition, please refer to FIG. 10. FIG. 10 is a diagram illustratinga depth map generation device 1000 capable of correcting occlusionaccording to a fifth embodiment of the present invention, wherein thedepth map generation device 1000 includes image capturers 102, 104, 106,107 and a depth map generator 108. But, the present invention is notlimited to the depth map generation device 1000 only including the imagecapturers 102, 104, 106, 107. As shown in FIG. 10, the image capturers104, 102 form a first image capture pair, the image capturers 104, 106form a second image capture pair, and the image capturers 104, 107 forma third image capture pair. Therefore, when a first depth mapcorresponding to the first image capture pair includes an occlusionregion and a second depth map corresponding to the second image capturepair does not include any occlusion region, the depth map generator 108of the depth map generation device 1000 can utilize the above-mentionedoperational principles of the depth map generator 108 of the depth mapgeneration device 100 to correct the occlusion region. In addition, asshown in FIG. 10, the depth map generator 108 can further fuse thesecond depth map corresponding to the second image capture pair and athird depth map corresponding to the third image capture pair togenerate a fused depth map, wherein the second depth map and the thirddepth map have different characteristics. For example, because a lengthof a baseline of the second image capture pair is not equal to a lengthof a baseline of the third image capture pair, a working range of thesecond depth map corresponding to the second image capture pair is alsonot equal to a working range of the third depth map corresponding to thethird image capture pair. Therefore, the depth map generator 108 canfuse the working range of the second depth map corresponding to thesecond image capture pair and the working range of the third depth mapcorresponding to the third image capture pair to generate the fuseddepth map.

In addition, please refer to FIG. 11. FIG. 11 is a diagram illustratinga depth map generation device 1100 capable of correcting occlusionaccording to a sixth embodiment of the present invention, wherein thedepth map generation device 1000 includes image capturers 102, 104, 105,106, 107 and a depth map generator 108. But, the present invention isnot limited to the depth map generation device 1100 only including theimage capturers 102, 104, 105, 106, 107. As shown in FIG. 11, the imagecapturers 104, 102 form a first image capture pair, the image capturers104, 105 form a second image capture pair, the image capturers 104, 106form a third image capture pair, and the image capturers 104, 107 form afourth image capture pair, wherein the first image capture pair and thesecond image capture pair belong to a first image capture pair group,and the third image capture pair and the fourth image capture pairbelong to a second image capture pair group. In sixth embodiment of thepresent invention, the depth map generator 108 can first fuse depth mapscorresponding to the first image capture pair group to generate a firstfused depth map, and fuse depth maps corresponding to the second imagecapture pair group to generate a second fused depth map, wherein qualityof the first fused depth map is better than quality of a first depth mapcorresponding to the first image capture pair and quality of a seconddepth map corresponding to the second image capture pair, and quality ofthe second fused depth map is better than quality of a third depth mapcorresponding to the third image capture pair and quality of a fourthdepth map corresponding to the fourth image capture pair. If the firstfused depth map includes a first occlusion region and a firstnon-occlusion region, the depth map generator 108 can utilize depthinformation of a valid area of the second fused depth map correspondingto the first occlusion region to correct the first occlusion regionaccording to the above-mentioned operational principles of the depth mapgenerator 108 of the depth map generation device 100.

To sum up, the depth map generation device capable of correctingocclusion utilizes a corresponding valid area of a second depth map ofat least two depth maps corresponding to at least two image capturepairs generated by the depth map generator to correct (or substitutefor) at least one occlusion region included in a first depth map of theat least two depth maps when the first depth map includes the at leastone occlusion region. Therefore, compared to the prior art, because thepresent invention utilizes an image capturer of the second image capturepair, wherein a visual field of the image capturer is not occluded, toobtain depth information of an object, the present invention caneffectively solve a problem of a depth map including an occlusionregion.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A depth map generation device capable ofcorrecting occlusion, comprising: at least two image capture pairscapturing a plurality of images; and a depth map generator coupled tothe at least two image capture pairs for generating a first depth mapand a second depth map according to the plurality of images, whereinwhen the first depth map comprises a first occlusion region and a firstnon-occlusion region, the depth map generator corrects the firstocclusion region according to the second depth map.
 2. The depth mapgeneration device of claim 1, wherein the first depth map corresponds toa first image capture pair of the at least two image capture pairs, thesecond depth map corresponds to a second image capture pair of the atleast two image capture pairs, and a first image capturer is comprisedin both the first image capture pair and the second image capture pair.3. The depth map generation device of claim 2, wherein a length of afirst baseline of the first image capture pair is equal to a length of asecond baseline of the second image capture pair.
 4. The depth mapgeneration device of claim 2, wherein when a length of a first baselineof the first image capture pair is not equal to a length of a secondbaseline of the second image capture pair, the depth map generatorfurther executes normalization on the second depth map to make adisparity of the first depth map match a disparity of the second depthmap.
 5. The depth map generation device of claim 2, wherein when a firstbaseline of the first image capture pair and a second baseline of thesecond image capture pair are not parallel to each other, the depth mapgenerator further executes geometric calibration on at least one of thefirst depth map and the second depth map.
 6. The depth map generationdevice of claim 5, wherein the geometric calibration is rotationcalibration.
 7. The depth map generation device of claim 2, wherein avisual field of a second image capturer of the first image capture paircorresponding to an object is occluded by a first object to make depthinformation of the first depth map corresponding to the object comprisesthe first occlusion region and the first non-occlusion region, and thedepth information of the first depth map corresponding to the firstocclusion region is an invalid area; and a visual field of a third imagecapturer of the second image capture pair corresponding to the object isnot occluded by the first object, and the depth map generator correctsthe first occlusion region according to depth information of the seconddepth map corresponding to the object.
 8. The depth map generationdevice of claim 2, wherein a visual field of a second image capturer ofthe first image capture pair corresponding to an object is occluded by afirst object to make depth information of the first depth mapcorresponding to the object comprises the first occlusion region and thefirst non-occlusion region, and the depth information of the first depthmap corresponding to the occlusion region is an invalid area; a visualfield of a third image capturer of the second image capture paircorresponding to the object is occluded by a second object to make depthinformation of the second depth map corresponding to the objectcomprises a second occlusion region and a second non-occlusion region,and the depth information of the second depth map corresponding to thesecond occlusion region is an invalid area; and the depth map generatorcorrects the second occlusion region and the first occlusion regionaccording to the first non-occlusion region and the second non-occlusionregion, respectively.
 9. The depth map generation device of claim 1,wherein each image capture pair of the at least two image capture pairsis composed of two image capturers, or an image capturer and a lightsource.
 10. The depth map generation device of claim 9, wherein thelight source is used for emitting structured light.
 11. A depth mapgeneration device capable of correcting occlusion, comprising: an imagecapture module comprising at least two image capture pairs, and eachimage capture pair of the at least two image capture pairs is composedof two image capturers, or an image capturer and a light source; and adepth map generator coupled to the image capture module, wherein when afirst depth map of at least two depth maps generated by the depth mapgenerator corresponding to the at least two image capture pairscomprises an occlusion region and a non-occlusion region, the depth mapgenerator utilizes a second depth map of the at least two depth maps tocorrect the occlusion region.
 12. The depth map generation device ofclaim 11, wherein the light source is used for emitting structuredlight, and when at least one image capture pair of the at least twoimage capture pairs is composed of the image capturer and the lightsource, the at least one image capture pair is used for capturing imagescomprising the structured light.
 13. The depth map generation device ofclaim 11, wherein the occlusion region is an invalid area of the firstdepth map formed by an object being occluded by another object.
 14. Adepth map generation device capable of correcting occlusion, comprising:at least two image capture pairs capturing a plurality of images; and adepth map generator coupled to the at least two image capture pairs forgenerating a first depth map and a second depth map according to theplurality of images, wherein the depth map generator further executes atleast one of correcting an occlusion region of the first depth mapaccording to the second depth map and fusing the first depth map and thesecond depth map to generate a fused depth map.
 15. A depth mapgeneration device capable of correcting occlusion, comprising: two imagecapture pair groups capturing a plurality of first images and aplurality of second images, respectively; and a depth map generatorcoupled to the two image capture pair groups for generating a pluralityof first depth maps and a plurality of second depth maps according tothe plurality of first images and the plurality of second imagesrespectively, and fusing the plurality of first depth maps and theplurality of second depth maps to generate a first fused depth map and asecond fused depth map, respectively; wherein when the first fused depthmap comprises a first occlusion region and a first non-occlusion region,the depth map generator further corrects the first occlusion regionaccording to the second fused depth map.